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Abstract 

Solving complicated problems in a contemporary knowledge-based society requires higher-order thinking (HOT). The 
most productive way to encourage development of HOT in students is through use of the Problem-based Learning 
(PBL) model. This model organizes learning by solving corresponding problems relative to study courses. Students are 
directed to develop HOT skills needed for problem-solving that are essential for their future professional activities. 
However, in order to promote effective HOT skills development of students in PBL learning environment, 
enhancement of the model is required. The model enhancement can be attained through specific computer-mediated 
assessment of learning of students. In this paper we introduce an innovative approach to complex, adaptive, and 
computer-mediated assessment of HOT skills development of individual students. The complexity of assessment is 
expressed by forming the combined assessments of HOT skills of different types. Adaptation of assessment to the 
process of HOT skills development is expressed by changes in an instructor's fixed assessments by crossing from one 
phase of PBL to another. Assessment adaptation is provided for individual students as for a study group. 
Computer-mediation of assessment is provided by a Computer Assessment Tool which promotes students’ HOT skills 
development and facilitates the assessment process for an instructor. The proposed coefficient of success of HOT skills 
development serves as an effective tool of analysis of this process. 

Keywords: Problem-Based Learning, Higher-Order thinking, Complex adaptive assessment, Computer-Mediated 
Assessment 

1. Introduction 

Solving complicated problems in a contemporary knowledge-based society requires higher-order thinking (HOT), 
which is provided as a result of developing mental skills in students. The most suitable model aimed at effective HOT 
skills development is the Problem-based Learning (PBL) model (Duch, Groh, Allen, 2001; Amador, Miles, Peters, 
2006; Bareli, 2006; Barret & Moore, 2010). According to this model, self-directed learning intends: problem 
determination; generation of a set of possible ideas for problem solving and a choice of a constructive idea; gaining 
needed information for problem solving, and problem solving actions. However, there are obstacles that make 
implementation of the PBL model difficult (Boud & Feletti, 1997; Schwartz, Mennin, and Webl, 2001). The main 
obstacles are that students are not fully prepared for learning on their own; they are not encouraged enough to develop 
HOT skills; and it is an extra-heavy workload for an instructor. 

The obstacles can be overcome through creation of a PBL environment which is favorable for HOT skills development. 
According to conducted research (Hannafin & Land, 1997; Jonassen, 1998; Bransford, Brown, and Cocking, 1999; 
Baden & Major, 2004; Pettigrew & Scholter, 2010), such an environment should be: 

• HOT skills-centered (an instructor must focus on HOT skills development for students); 

• Student-centered (an instructor should encourage student interest in problem solving and critical thinking); 

• Student adaptive (adaptation to the given student though testing or monitoring his (her) performance); 

• Assessment centered (an instructor should provide high quantity and quality of assessments while stimulating 
students for HOT skills development); 

• Computer-supported (using computer tools to decrease an extra-heavy workload for an instructor). 

The central role in the creation of a PBL environment favorable for HOT is given to an instructor. The instructor should 
guide, foster, facilitate, and stimulate HOT skills development. A specific approach to computer-mediated assessment 
of HOT skills development in students can promote realization of this role of an instructor. 
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The approach should be based on the following assessment principles (Macdonald & Savin-Baden, 2004; Brookhart, 
2010): specify clearly and exactly the matter of assessment; assessment is an instruction tool and not an evaluation 
mechanism; an assessment should keep learners motivated and involved; flexibility and complexity of assessment 
should be provided; assessment should be relevant to desired learning outcomes, and assessment should be a guided, 
adaptive, formative, and summative process. 

The proposed systematic approach is aimed at computer-mediated complex adaptive assessment of HOT skills 
development of individual students and a class. The systematic approach should be ensured through reliability, validity, 
and usability of assessment. 

2. Related Research 

The goal of the proposed approach is enhancing a PBL model through computer-mediated assessment of HOT skills 
development in students. Organized correctly, the assessment process drives developing HOT skills needed for 
problem solving. The approaches, methods, and tools reviewed here are examined relative to various aspects of this 
process. 

Knight (2000) pointed to the need of a systematic approach to the assessment of students’ learning which provides 
reliability, validity, and usability. Moallem (2007) considers the process of evaluation relative to the following stages: 
initial assessment, progress assessment, and product assessment. Ma and Zhou (2000) suggested a fuzzy set approach 
to assess outcomes of student learning. These authors claim that their aim is encouraging students to participate in the 
whole learning process. Tillema (2010) is quite right contending that the function of formative assessment is to 
promote learning. Papanastasiou (2014) defined adaptive assessment as a type of assessment that is set for each student 
taking into account an assessment of his (her) previous performance. This allows an assessment to be more accurate of 
individual ability. 

Savery (2006) affirms that the goals of PBL are knowledge building and adoption of a problem-solving process. Hence, 
student progress towards the goals of PBL should be assessed. Lovie-Kitchin (2001) rightly claims assessment 
methods should be congruent with the PBL process. Macdonald and Savin-Baden (2004) stressed the need for specific 
assessment methods for PBL. The authors note assessment should reflect the practical orientation of PBL. 
Consequently, HOT skills should be assessed. Brookhart (2010) asserts that holding students accountable for HOT 
through assessment increases student motivation and improves the results of learning. Student assessment is realized 
through reflection on built knowledge. 

Alam (2014) emphasized the importance of using technology in teaching and learning. The author demonstrated that 
the use of computer-based assessments can improve traditional methods of assessment and learning outcomes. Ma 
(2002) developed a decision support system for assessing PBL. Assessment criteria and their weights are based on the 
use of mathematical decision methods. 

The analysis of publications above shows that no a straightforward systematic approach exists to enhancing the PBL 
model due to computer mediated complex adaptive assessment of HOT skills development. The proposed systematic 
approach will promote productive HOT skills development due to the: 

• Reliability, validity, and usability of assessment 

• Stimulation of HOT skills development in students through complex and adaptive assessment 

• Flexibility, responsiveness, and versatility of assessment caused by computer mediation 

• Capacity of changing a set of HOT skills for assessment 

• Facilitation of instructor work 

• Management of HOT skills development 

3. The Systematic Approach to Computer-Mediated Assessment of HOT Development 

The aim of the proposed systematic approach is realizing complex, adaptive, and computer mediated assessment of 
developing HOT skills needed for problem solving. Complexity of assessment is provided by forming the combined 
assessments of HOT skills of different types. Adaptation of assessment to the process of HOT skills development is 
expressed by changes in fixed instructor assessments by crossing from one phase of PBL to another. Computer 
mediation facilitates the assessment process for an instructor. 

There are two distinct types of HOT skills: analytical and creative thinking skills. Analytical or logical thinking 
realizes critical thinking and help select the best alternative. The analytical thinking skills are: ordering, comparing, 
contrasting, evaluating, and selecting. Creative thinking skills are needed for problem solving. The creative thinking 
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skills are efficiency (producing many ideas), flexibility (producing a broad range of ideas), originality (producing 
uncommon ideas), and elaboration (developing ideas) (Bednarz, 2011; Cottrell, 2011; Cottrell, 2013). 

The systematic approach ensures reliability, validity, and usability of assessment. Reliability of student assessment is 
attained through assessing each skill and their aggregates. Validity of assessment is provided by the change of fixed 
assessment of skills on the basis of learning progress. Usability is caused by the simplicity and transparency of the 
assessment process. The approach intends developing a method of complex adaptive assessment of HOT skills, and a 
Computer Assessment Tool (CAT) promoting HOT skills development of students and facilitating the assessment 
process for an instructor. 

3.1 The method of complex dynamic assessment of HOT skills 

The proposed method consists of: setting fixed initial assessments of HOT skills of different types for the first study 
phase of a course based on the PBL model; assessment of the individual skills; realizing combined assessments for 
each type of HOT skills; and forming adaptive assessments for the following study phases of a course on the basis of 
analysis of determined coefficients of success of HOT skills development. Consequently, the method is realized 
through a sequence of the steps: 

1. Setting fixed initial assessments of HOT skills 

2. Assessing development of analytical thinking skills 

3. Assessing development of creative thinking skills 

4. Forming combined formative assessments of HOT skills development 

5. Determining a coefficient of success of HOT skills development 

6. Developing Adaptive Assessments 

A. Setting fixed initial assessments of HOT skills 

The fixed initial assessments are set for each HOT skill as for each type of skill set. Difference of the fixed initial 
assessments of the skills expresses the preference of an instructor in the development of certain skills during the first 
study stage of a course based on PBL. An example of setting the fixed initial assessments is represented by Table 1. The 
maximum fixed initial assessments are set for the “Contrasting" skill from the “Analytical” type and the “Originality” 
skill from the “Creative” type. Hence, these skills have the greatest need for development. 

Table 1. Fixed initial assessments of HOT skills 


The type of thinking skills 

Skills name 

The fixed initial 

assessments (%) 


ordering 

8 


comparing 

6 

Analytical (40%) 

contrasting 

14 


evaluating 

7 


selecting 

5 


efficiency 

16 


flexibility 

12 

Creative (60%) 

originality 

22 


elaboration 

10 


B. Assessing development of analytical thinking skills 

An instructor prepares a complete list of questions which concern a study course by a student group. The questions of 
the list should examine analytical thinking skills after the first study stage of a course based on PBL. Next, the 
instructor composes individual lists of control questions on the basis of the complete list. An individual list can contain 
some questions for examination of a certain skill. The control questions from an individual list should provide 
examination of all analytical skills. 

A student’s skill development is assessed by an instructor through the answers given by a student to control questions 
from the individual list. An instructor assessment of a student answer to a question varies from zero to the fixed initial 
assessment of analytical skill corresponding to a question. If a student cannot answer a question, or an answer is 
incorrect, then an instructor assessment is equal to zero. If a student answers a question corresponding to certain skill 
correctly, an instructor assessment is equal to fixed initial assessment of corresponding analytical skill. It means he or 
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she has developed this skill. If a student answer is not full, an instructor sets suitable assessment. As a result, an 
assessment of a student is made. Assessments of other students from a study group are set analogously. The obtained 
assessments allow an instructor to assess lack of developing students’ analytical skills through comparison of the 
results of student answers with fixed initial assessments. An example of assessment of developing analytical skills in 
students is represented by Table2. Formative assessments of analytical skills development in student Si, s 2 , and S 3 are 11, 
21 , and 35, accordingly. 

Table 2. Formative assessments of analytical skills development in students 


The Questions The fixed initial 
assessments of 
analytical skills 

Student Si 
assessment 

Student S 2 
assessment 

Student S 3 
assessment 

qn 

Ordering 4% 

0 

4 

3 

Ql2 

Ordering 4% 

0 

4 

3 

q2i 

Comparing 6 % 

3 

3 

5 

qsi 

Contrasting 7% 

0 

7 

7 

q32 

Contrasting 7% 

0 

0 

7 

q« 

Evaluating 7% 

3 

3 

6 

qsi 

Selecting 5% 

5 

0 

4 


C. Assessing development of creative thinking skills 

An instructor prepares a complete list of control problems for solving by study group students. The problems from the 
list should examine creative thinking skills. Next, the instructor composes individual lists of control problems based on 
the complete list. The aggregate of the control problems of an individual list should examine development of all 
creative skills. Development of student skills is assessed by an instructor during problem solving from the individual 
list. 

An instructor assessment of a student creative skill varies from zero to the fixed initial assessments of a creative skill. If 
a student didn’t reveal presence of a certain creative skill owing to solving individual control problems, an instructor 
assessment is equal to zero. It means the student did not develop this skill. If a student solved an individual control 
problem partially, an instructor sets suitable assessment. If a student revealed the presence of a certain creative skill 
owing to solving individual control problems, an instructor assessment is equal to the fixed initial assessments of the 
skill. It means the student developed this skill. As a result, a formative assessment of creative thinking skills in a 
student is made. Formative assessments of developing creative skills in other students in a study group are determined 
analogously. An example of assessment of developing creative skills in students is represented by Table 3. Formative 
assessments of the creative skills development in student s b s 2 , and s 3 are 16, 22, and 48, accordingly. The obtained 
formative assessments allow for assessing lack of students’ creative skills development through comparison of the 
results of an individual student’s solving of control problems with fixed initial assessments. 

Table 3. Formative assessments of creative skills development in students 


The fixed initial 
assessments of creative 

skills 

Student Si 
assessment 

Student S 2 
assessment 

Student S 3 

assessment 

Efficiency 16% 

6 

4 

16 

Flexibility 12% 

0 

12 

12 

Originality 22% 

0 

0 

20 

Elaboration 10% 

10 

6 

0 


D. Forming combined formative assessments of HOT skills development 

Combined assessments of developing HOT skills are determined as a result of summation of corresponding 
assessments of analytical and creative skills. An example of the combined assessments based on data from Table 2 and 
Table3 are presented in Table 4. 


Published by Sciedu Press 


228 


ISSN 1927-6044 E-ISSN1927-6052 





www.sciedu.ca/ijhe 


International Journal of Higher Education 


Vol. 4, No. 1; 2015 


Table 4. Combined formative assessments of the HOT skills development 



s, 

s 2 

S 3 

Formative assessments of developing 
analytical skills 

11 

21 

35 

Formative assessments of developing 
creative skills 

16 

22 

48 

Combined formative assessments of 
developing HOT skills 

27 

43 

83 


E. Determining a coefficient of success in HOT skills development 

Analysis of HOT skills development in students is based on comparison of formative assessments of HOT skills 
development and fixed assessments of skills. Such comparison is realized by using a formula: 

5 (s;) = (g(sO - g) / g , - 1 < 5(sj) < 0 ( 1 ) 

where 

g(sd is a formative assessment of the development of the certain HOT skill (some or all skills of the defined type) in 
student sg g is a fixed assessment of all HOT skills of defined type. It is the sum of the fixed assessments of particular 
skills; 6(Sj) is a coefficient of success of HOT skills development in student s,-. 

The value of the coefficient of success characterizes HOT development relative to separate skills and their 
combinations. Thereby, analysis of results of HOT skills development should be realized by examination of values of 
the coefficient of success. The proposed coefficient of success of HOT skills development serves as an effective tool of 
quantitative and qualitative analysis of developing thinking skills. 

The fixed assessment of the skill “Selecting" is 5%, and a fixed assessment of all HOT skills of “Analytical” type is g= 
40% (Tablel). The student Si developed the skill “Selecting" (Table 2). Hence, his formative assessment relative to this 
skill is g(si) =5%. Then, the value of the coefficient of success of student Si relative to this skill according to formula (1) 
is 5 S = - 0.875. 

The student S 3 has the most formative assessment of analytical skills development in a study group. The dominant part 
of skills developed by student S 3 is constituted by the skills “Contrasting, Ordering, and Evaluating". The formative 
assessment is gfe) = 35% (Table 2). The fixed assessment of all HOT skills of the “Analytical” type is g= 40% 
(Tablel). Then, the coefficient of success of the development of analytical skills by student S 3 according to formula (1) 
is 5 a (s 3 ) = - 0.125. It allows the conclusion that student S 3 has an inessential lack of analytical skills since: 6 (S 3 ) = - 
0.125 and the maximal value of the coefficient of success of developing skills equals zero. 

The fixed assessment of developing creative thinking skills is g= 60 (Tablel). The student S 3 has the most assessment of 
developing creative skills in a study group. The assessment is g(s 3 ) = 48% (Table 3). Then, the coefficient of success of 
developing creative skills in student s 3 according to formula (1) is 6 c (s 3 ) = - 0.2. Comparison of the coefficient 5 A (s 3 ) = 

- 0.125 with the coefficient 5 C (S 3 ) = - 0.2 allows to conclude that student S 3 has had more success in development of 
analytical skills. 

The fixed assessment of developing HOT skills in a student is 100% (Tablel). Combined formative assessment of the 
development of the analytical skills and the creative skills in the student S 3 is 83% (Table 4). Then, the coefficient of 
success of the HOT skills development in student s 3 according to formula (1) is 5 (s 3 ) = - 0.17. It allows the conclusion 
that the student s 3 has an inessential lack of development of HOT skills since the maximal value of the coefficient of 
success of HOT skills development equals zero. 

F. Developing Adaptive Assessments 

Examination results analysis of HOT skills development with the help of the determined coefficient of success reveals 
a lack of development certain skills in individual students as a study group. It reflects the need for assessment 
adaptation in the process of HOT skills development for individual students as for a study group. Adaptation of 
assessment is realized by change of the fixed HOT skills assessments by crossing from one phase study to another. 

Changing the fixed assessments is carried out for individual students or for a study group in the following way. If 
examination results revealed a lack of the certain skill development for one or some students then fixed assessment of 
this skill is increased by an instructor. If the value coefficient of success of developing the defined type of skill is low, 
then fixed assessment of this skill type is increased by an instructor. Fixed assessment of a skill type having a larger 
value coefficient of success of developing skills is decreased. It induces the students to develop the needed skills. 
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3.2 A Computer Assessment Tool for HOT skills development 

The aim of the CAT is to promote students’ HOT skills development through facilitation of an assessment process for 
an instructor. On the basis of the data received from an instructor the CAT forms the next outcomes: 

• Formative assessments of the analytical and creative skills development of individual students for a certain 
phase of a course study 

• Average formative assessments of the analytical and creative skills development of individual students as a 
result of a course study 

• Combined formative assessments of developing HOT skills in individual students 

• The values of the coefficient in success developing certain type skills of individual students 

• The values of coefficient in success developing the HOT skills of individual students 

• Average formative assessments of the analytical and creative skills development of a class relative to a certain 
phase of course study and a course 

• Average formative assessments of the HOT skills development of a class relative to a study phase and a course 

• The thinking skills that revealed poor and improved development relative to every student 

• The thinking skills that revealed poor and improved development by all students 

• The best student and the worst student relative to HOT skills development 

• The list of advanced students. 

An instructor and students interact with each other and with the CAT. The interaction between an instructor and 
students is as follows. An instructor leads PBL of the students. He also offers control questions and problems for 
students in order to assess analytical and creative skills development. The students interact with an instructor during 
PBL and present answers for the control questions and the results of problem solving. During the interactions between 
an instructor and the CAT, an instructor fills in the data on the form for assessment of analytical and the creative skill 
development (similarly by the Table2 and the Table3). The data is: 

• a list of the analytical skills and fixed assessments of the analytical skills development 

• a list of control questions 

• the fixed assessments for the control questions 

• the results of the student answers for the control questions 

• a list of the creative skills and the fixed assessments of creative skills development 

• the results of developing the corresponding creative skills by the students due to the problem solving 

The CAT gives to an instructor the above-mentioned outcomes. During the interactions between a student and the CAT, 
a student inquires of the CAT about assessments of his (her) HOT skills development. The CAT gives to a student: 
formative assessments of analytical and creative skills development as a result of a study phase of the course; average 
formative assessments of the analytical and creative skills development as a result of course study; the thinking skills 
that had poor and improved development. 

4. Conclusion and Future Work 

This paper has introduced a straightforward systematic approach to computer-mediated assessment of developing HOT 
skills needed for problem solving. The Computer Assessment Tool is used for assessing HOT skills development in 
students. The approach promotes effective development of HOT skills in students, motivates active learning, and 
facilitates use of PBL model by an instructor and students. 

The approach provides complexity, adaptability, flexibility, responsiveness, and versatility of assessing HOT skills 
development. Complexity of assessment is expressed in forming the combined assessments of HOT skills of different 
types. Adaptation of assessment to HOT skills development of individual students and a study group is represented by 
changes in an instructors fixed assessments of students’ HOT skills by crossing from one phase of PBL to another. 
Change of fixed skills assessments is realized through analysis of values of the proposed coefficient of success of HOT 
skills development. Flexibility, responsiveness, and versatility are reached due to use of a CAT. 

Further research will be directed towards enhancement of the proposed approach, analysis of its efficiency as a result of 
practical use, and creation of an intelligent tool for management of HOT skills development. 
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